ENG-234

Any new visualisations based on lots of lots of data??? @Abby010 and make sure you are copying over and synthesizing the details I have earlier.

Any new visualisations based on lots of lots of data??? @Abby010 and make sure you are copying over and synthesizing the details I have earlier.

Not yet, I starting getting some dependency errors so I needed to fix that but I have started working with lots of data.

New video links have been added with more data in the description

Add comments rather than editing spec for that stuff.

@Abby010 can you:

1. Get the CI working
2. Start integrating this into the js-logger src code under a new class object Tracer.
3. Plan out integration into Polykey-CLI for testing.
4. Get @aryanjassal or @tegefaulkes to review to see how it can hook into object lifecycles and node native async contexts.

I think it's important for testing why seed nodes are crashing.

Also why use -+ and not \?

Also you should be assigning yourself.

Start adding tests too.

And benchmarks.

I want a plan as of next cycle.

Make sure that the span structure can be streamed into an incremental jsonl format. This is what we will be using to then stream visualise on the browser. @shafiqihtsham can weigh in if you talk to him about it.

But I want you to write out a plan for implementation in Polykey-CLI by today.

Tracing Integration Plan

Phase 1: Integrate Tracing into js-logger as a First-Class Component

• Introduce a Tracer class that encapsulates all span-related logic.
• Replace the standalone openSpan, closeSpan functions with methods on this class.
• Support structured span creation: startSpan, endSpan, and traced wrappers with context tracking.
• Provide methods for exporting span data:
  • getActiveSpans()
  • flush()
  • getTraceJSON()

Add Incremental Span Streaming to JSONL

• Each span, when closed, is written as a JSON object to a .jsonl file (e.g., spans.jsonl).

• Enable atomic streaming via fs.appendFileSync or streaming writers.

• Supports real-time debugging and integration with external tools.

• Update the current TUI visualizer to consume spans from the new Tracer instance.

This phase formalizes span management, embeds JSONL observability, and prepares for broader CLI and system integration.

Phase 2: Integrate Tracer with Polykey CLI and Lifecycle Events

• Initialize and embed the Tracer into the Polykey CLI runtime.
• Use Tracer.traced() to instrument top-level CLI commands and subcommands.
• Extend tracing into object lifecycle tracking: vaults, daemons, agents, etc.
• Hook into Node.js async context (AsyncLocalStorage) to propagate spans implicitly across async calls and promise chains.
• Stream span data from real Polykey CLI runs to the .jsonl file.
• Analyze and visualize spans in the TUI to confirm accurate instrumentation.
• Use the streamed data for:
  • Automated debugging
  • Replay analysis
  • Bug discovery

This phase validates the integration by capturing real CLI behaviors and preparing for long-term observability and tracing.

Your phase 2 doesn't mention https://github.com/MatrixAI/js-async-init at all. You both need to talk about the implementation plan with @tegefaulkes since I think you're missing some contextual details critical to understanding how tracing can be applied to Polykey's runtime.

Phase 2 Plan: Polykey CLI Tracing Integration

Objective

Instrument the Polykey CLI runtime using the Tracer to support structured, memory-safe, and streamable observability of system behavior — with minimal invasiveness and full reversibility.

1. Embed Tracer into Polykey CLI Runtime

• Inject a Tracer instance during CLI bootstrap.
• Pass it into relevant components (e.g., Agent, VaultManager) via dependency injection or service context.
• Ensure span data is optional and tracing is enabled only via flags like --enable-tracing.

2. Hook into Lifecycle Events via js-async-init

Utilize @AsyncInit decorators and structured lifecycle events defined in the js-async-init library.

For async-initialized components like NodeConnectionManager, VaultManager, Daemon, etc., hook into their lifecycle using:

• EventStart, EventStop
• EventCreate, EventDestroy

Instrumentation Pattern:

nodeManager.addEventListener(events.NodeConnectionManagerStart.name, () => {
  nodeManager.span = parentSpan.getChild('NodeManagerStart');
});

nodeManager.addEventListener(events.NodeConnectionManagerStop.name, () => {
  nodeManager.span.end();
});

3. Adopt Span Object Model (No Global Tracking)

Replace global tracer calls with direct Span object usage.

Example:

const span = parentSpan.getChild('VaultFetch');
span.event('RPC Request Sent');
span.end();

Track spans via objects, not IDs. Avoid propagating span IDs through business logic.

4. Stream Output Instead of Writing Directly

Refactor Tracer to emit to a writable stream (WritableStream, PassThrough, etc.).

Consumers of the stream (e.g., file writer, stderr, custom logger) decide how to handle the data.

CLI Support:

--trace-output spans.jsonl
--trace-output stderr
--trace-fd=3    # for advanced use

5. Memory Management: No activeSpans, Use WeakRefs

Remove Map<string, Span> used for global span tracking.
Span objects should be ephemeral and garbage collected naturally.

Use WeakRef where necessary for parent-child relationships.
Prevent span graphs from being retained in memory permanently.

6. Implement Observable Tracing Events

Tracer emits observable events:

span:start
span:end

Useful for real-time debugging, TUI enhancements, or plugin systems.

7. Testing Strategy: Use fast-check for Randomized Span Behavior

Replace Math.random() in test harnesses with fast-check property-based fuzzing.

Supports reproducible failures and automated test shrinking for minimal failing cases.

8. Visual Debugging with TUI (as-is)

No changes needed to TUI.

Continues to read from the .jsonl file.

9.Leveraging js-async-init's Lifecycle Contracts

Use lifecycle annotations from js-async-init to instrument long-lived services and ephemeral agents with start/stop or create/destroy lifecycles. The Tracer will hook into lifecycle phases without modifying the business logic directly.

Each lifecycle boundary maps naturally to a span (e.g., VaultManagerStart, DaemonCreate, NodeConnectionDestroy).
This ensures:

1. Lifecycle spans align with runtime semantics.
2. Tracing remains declarative and reversible.
3. No modification to object constructors or internals.

Some things to review:

• https://github.com/tc39/proposal-async-context
• Review what runtimes support it already (node already does)
• The "span" object should be more implicit, it explore the idea of using WeakMap and symbols for synthetic identifiers.
• Right now committing to an integration path doesn't make sense, try monkey patching at the Polykey runtime itself, and just output logs and visualising that separately.

Reviewing https://github.com/tc39/proposal-async-context shows that:

1. https://github.com/tc39/proposal-async-context AsyncContext integration with web APIs w3c/tpac2023-breakouts#39 https://nodejs.org/api/async_context.html all indicate that this feature isn't available widely in all of JS.
2. If we were to use async context, the logger would be forced to only work in nodejs runtime.
3. This is fine for debugging PK, but not so much crossing into browsers and we want this library to be generic.
4. I realise that we should be able to "approximate" this feature by using our decorators, we have js-contexts https://github.com/matrixai/js-contexts that brings together a generic way of creating decorators providing context. It should be possible to use that along with js-async-init to create some sort of more implicit system, that can at first use the node's async context when available, or use something a bit more manual when it is not available (like a combination of weakmap and symbols).

@Abby010 feedback?

Key thing about this is that our spans don't necessarily end to be useful.

Reviewing https://github.com/tc39/proposal-async-context shows that:

1. https://github.com/tc39/proposal-async-context AsyncContext integration with web APIs w3c/tpac2023-breakouts#39 https://nodejs.org/api/async_context.html all indicate that this feature isn't available widely in all of JS.
2. If we were to use async context, the logger would be forced to only work in nodejs runtime.
3. This is fine for debugging PK, but not so much crossing into browsers and we want this library to be generic.
4. I realise that we should be able to "approximate" this feature by using our decorators, we have js-contexts https://github.com/matrixai/js-contexts that brings together a generic way of creating decorators providing context. It should be possible to use that along with js-async-init to create some sort of more implicit system, that can at first use the node's async context when available, or use something a bit more manual when it is not available (like a combination of weakmap and symbols).

@Abby010 feedback?

Re: async_context Proposal
I’ve reviewed tc39/proposal-async-context and the Node.js async_hooks API. While these are promising for implicit span propagation, they’re not yet cross-platform.

Agreed for now, relying solely on Node-specific behavior would limit portability. Since js-logger should remain runtime-agnostic, we shouldn’t hard-bind to Node.js-only primitives.

Re: Span Should Be More Implicit
Makes sense. Right now, spans are passed explicitly or manually linked via getChild(). I'm exploring using:

1. Symbols to tag async flows.
2. WeakMap to manage span-local state for async resources as suggested
3. Decorators (js-contexts + js-async-init) to set up span context implicitly without forcing manual tracing at every step.

This would allow something like:

@Traceable
async function syncVault() {
  // span is implicitly available via decorator-generated context
  currentSpan.event('Vault Sync Triggered');
}

Re: Monkey Patching in Polykey Runtime
That aligns with the Phase 2 goal: non-invasive, reversible tracing.
Before embedding tracing into the CLI codebase, We will monkey-patch lifecycle events in runtime modules (like VaultManager, NodeConnectionManager) and observe the emitted span logs via JSONL/TUI.

This will let us test the conceptual model before formalizing APIs or integrations.

Re: js-contexts + js-async-init Hybrid
I’m looking into combining js-async-init lifecycle events with js-contexts to simulate AsyncLocalStorage-like behavior generically.
Idea is to support both:
Native async context via Node.js hooks.
Fallback shim using js-contexts and WeakRefs where native async context isn't available.

The original plan was for @aryanjassal to start using this in Polykey regardless of the merge. But if that's too difficult. Start merging this in now incrementally and release new ones so that PK can start testing it. This week it has to be done. There's no more time to be allocated to this.

@aryanjassal you now take over this PR to force it to be merged and released.

@aryanjassal start squashing and merge into a patch version, then use it in a new PR in Polykey-CLI.

Due to not having sufficient time for the handover from abby, there might be some minor issues which might make it to staging. This might see a lot of iteration as well. All the iteration will be done in patch updates which will be used by Polykey CLI.

The js-async-init package will be monkey-patched first to get kind of a demo of the library, and iteration will be made to this library until we have reached a good state to analyse this in-depth.

I've cleaned up the codebase a little bit, so this can now be merged to staging. I will need to go over some things with abby about the future plans of the tracer, but until then I can start testing out monkey-patching.

It doesn't make much sense to make a PR for this as the change will be in node modules, which is local-only and making a PR will not make a difference. I will however make a branch for this and push up a PR if any significant changes are made in there.